STAnet: A Spatiotemporal Attention Network for Decoding Auditory Spatial Attention From EEG.

OBJECTIVE: Humans are able to localize the source of a sound. This enables them to direct attention to a particular speaker in a cocktail party. Psycho-acoustic studies show that the sensory cortices of the human brain respond to the location of sound sources differently, and the auditory attention itself is a dynamic and temporally based brain activity. In this work, we seek to build a computational model which uses both spatial and temporal information manifested in EEG signals for auditory spatial attention detection (ASAD). METHODS: We propose an end-to-end spatiotemporal attention network, denoted as STAnet, to detect auditory spatial attention from EEG. The STAnet is designed to assign differentiated weights dynamically to EEG channels through a spatial attention mechanism, and to temporal patterns in EEG signals through a temporal attention mechanism. RESULTS: We report the ASAD experiments on two publicly available datasets. The STAnet outperforms other competitive models by a large margin under various experimental conditions. Its attention decision for 1-second decision window outperforms that of the state-of-the-art techniques for 10-second decision window. Experimental results also demonstrate that the STAnet achieves competitive performance on EEG signals ranging from 64 to as few as 16 channels. CONCLUSION: This study provides evidence suggesting that efficient low-density EEG online decoding is within reach. SIGNIFICANCE: This study also marks an important step towards the practical implementation of ASAD in real life applications.

Auditory Attention Detection with EEG Channel Attention.

Auditory attention detection (AAD) seeks to detect the attended speech from EEG signals in a multi-talker scenario, i.e. cocktail party. As the EEG channels reflect the activities of different brain areas, a task-oriented channel selection technique improves the performance of brain-computer interface applications. In this study, we propose a soft channel attention mechanism, instead of hard channel selection, that derives an EEG channel mask by optimizing the auditory attention detection task. The neural AAD system consists of a neural channel attention mechanism and a convolutional neural network (CNN) classifier. We evaluate the proposed framework on a publicly available database. We achieve 88.3% and 77.2% for 2-second and 0.1-second decision windows with 64-channel EEG; and 86.1% and 83.9% for 2-second decision windows with 32-channel and 16-channel EEG, respectively. The proposed framework outperforms other competitive models by a large margin across all test cases.

Auditory Attention Detection via Cross-Modal Attention.

Humans show a remarkable perceptual ability to select the speech stream of interest among multiple competing speakers. Previous studies demonstrated that auditory attention detection (AAD) can infer which speaker is attended by analyzing a listener's electroencephalography (EEG) activities. However, previous AAD approaches perform poorly on short signal segments, more advanced decoding strategies are needed to realize robust real-time AAD. In this study, we propose a novel approach, i.e., cross-modal attention-based AAD (CMAA), to exploit the discriminative features and the correlation between audio and EEG signals. With this mechanism, we hope to dynamically adapt the interactions and fuse cross-modal information by directly attending to audio and EEG features, thereby detecting the auditory attention activities manifested in brain signals. We also validate the CMAA model through data visualization and comprehensive experiments on a publicly available database. Experiments show that the CMAA achieves accuracy values of 82.8, 86.4, and 87.6% for 1-, 2-, and 5-s decision windows under anechoic conditions, respectively; for a 2-s decision window, it achieves an average of 84.1% under real-world reverberant conditions. The proposed CMAA network not only achieves better performance than the conventional linear model, but also outperforms the state-of-the-art non-linear approaches. These results and data visualization suggest that the CMAA model can dynamically adapt the interactions and fuse cross-modal information by directly attending to audio and EEG features in order to improve the AAD performance.

A novel motor imagery EEG decoding method based on feature separation.

Objective. Motor imagery electroencephalography (EEG) decoding is a vital technology for the brain-computer interface (BCI) systems and has been widely studied in recent years. However, the original EEG signals usually contain a lot of class-independent information, and the existing motor imagery EEG decoding methods are easily interfered by this irrelevant information, which greatly limits the decoding accuracy of these methods.Approach. To overcome the interference of the class-independent information, a motor imagery EEG decoding method based on feature separation is proposed in this paper. Furthermore, a feature separation network based on adversarial learning (FSNAL) is designed for the feature separation of the original EEG samples. First, the class-related features and class-independent features are separated by the proposed FSNAL framework, and then motor imagery EEG decoding is performed only according to the class-related features to avoid the adverse effects of class-independent features.Main results. To validate the effectiveness of the proposed motor imagery EEG decoding method, we conduct some experiments on two public EEG datasets (the BCI competition IV 2a and 2b datasets). The experimental results comparison between our method and some state-of-the-art methods demonstrates that our motor imagery EEG decoding method outperforms all the compared methods on the two experimental datasets.Significance. Our motor imagery EEG decoding method can alleviate the interference of class-independent features, and it has great application potential for improving the performance of motor imagery BCI systems in the near future.

Online compensation detecting for real-time reduction of compensatory motions during reaching: a pilot study with stroke survivors.

BACKGROUND: Compensations are commonly observed in patients with stroke when they engage in reaching without supervision; these behaviors may be detrimental to long-term functional improvement. Automatic detection and reduction of compensation cab help patients perform tasks correctly and promote better upper extremity recovery. OBJECTIVE: Our first objective is to verify the feasibility of detecting compensation online using machine learning methods and pressure distribution data. Second objective was to investigate whether compensations of stroke survivors can be reduced by audiovisual or force feedback. The third objective was to compare the effectiveness of audiovisual and force feedback in reducing compensation. METHODS: Eight patients with stroke performed reaching tasks while pressure distribution data were recorded. Both the offline and online recognition accuracy were investigated to assess the feasibility of applying a support vector machine (SVM) based compensation detection system. During reduction of compensation, audiovisual feedback was delivered using virtual reality technology, and force feedback was delivered through a rehabilitation robot. RESULTS: Good classification performance was obtained in online compensation recognition, with an average F1-score of over 0.95. Based on accurate online detection, real-time feedback significantly decreased compensations of patients with stroke in comparison with no-feedback condition (p < 0.001). Meanwhile, the difference between audiovisual and force feedback was also significant (p < 0.001) and force feedback was more effective in reducing compensation in patients with stroke. CONCLUSIONS: Accurate online recognition validated the feasibility of monitoring compensations using machine learning algorithms and pressure distribution data. Reliable online detection also paved the way for reducing compensations by providing feedback to patients with stroke. Our findings suggested that real-time feedback could be an effective approach to reducing compensatory patterns and force feedback demonstrated a more enviable potential compared with audiovisual feedback.

Real-Time Detection of Compensatory Patterns in Patients With Stroke to Reduce Compensation During Robotic Rehabilitation Therapy.

OBJECTIVES: Compensations are commonly employed by patients with stroke during rehabilitation without therapist supervision, leading to suboptimal recovery outcomes. This study investigated the feasibility of the real-time monitoring of compensation in patients with stroke by using pressure distribution data and machine learning algorithms. Whether trunk compensation can be reduced by combining the online detection of compensation and haptic feedback of a rehabilitation robot was also investigated. METHODS: Six patients with stroke did three forms of reaching movements while pressure distribution data were recorded as Dataset1. A support vector machine (SVM) classifier was trained with features extracted from Dataset1. Then, two other patients with stroke performed reaching tasks, and the SVM classifier trained by Dataset1 was employed to classify the compensatory patterns online. Based on the real-time monitoring of compensation, a rehabilitation robot provided an assistive force to patients with stroke to reduce compensations. RESULTS: Good classification performance (F1 score > 0.95) was obtained in both offline and online compensation analysis using the SVM classifier and pressure distribution data of patients with stroke. Based on the real-time detection of compensatory patterns, the angles of trunk rotation, trunk lean-forward and trunk-scapula elevation decreased by 46.95%, 32.35% and 23.75%, respectively. CONCLUSION: High classification accuracies verified the feasibility of detecting compensation in patients with stroke based on pressure distribution data. Since the validity and reliability of the online detection of compensation has been verified, this classifier can be incorporated into a rehabilitation robot to reduce trunk compensations in patients with stroke.